Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add db.utils.DBMAsyncJob #9656

Merged
merged 1 commit into from
Jul 8, 2021
Merged

Add db.utils.DBMAsyncJob #9656

merged 1 commit into from
Jul 8, 2021

Conversation

djova
Copy link
Contributor

@djova djova commented Jul 8, 2021

What does this PR do?

Adds a new DBMAsyncJob class which will be used by the postgres and mysql checks to schedule asynchronous tasks that need to run independently of the main check.

Follow-up PRs:

Motivation

Standardize approach for doing work asynchronously using threads for DBM.

Review checklist (to be filled by reviewers)

  • Feature or bugfix MUST have appropriate tests (unit, integration, e2e)
  • PR title must be written as a CHANGELOG entry (see why)
  • Files changes must correspond to the primary purpose of the PR as described in the title (small unrelated changes should have their own PR)
  • PR must have changelog/ and integration/ labels attached

@codecov
Copy link

codecov bot commented Jul 8, 2021

Codecov Report

Merging #9656 (064a664) into master (6d5a08c) will increase coverage by 0.03%.
The diff coverage is 90.44%.

Flag Coverage Δ
active_directory 100.00% <ø> (ø)
activemq_xml 82.31% <ø> (ø)
aerospike 86.97% <ø> (+0.36%) ⬆️
airflow 89.94% <ø> (ø)
amazon_msk 87.94% <ø> (ø)
ambari 86.98% <ø> (ø)
apache 95.39% <ø> (ø)
aspdotnet 93.87% <ø> (ø)
azure_iot_edge 82.01% <ø> (ø)
btrfs 82.91% <ø> (ø)
cacti 83.95% <ø> (ø)
cassandra_nodetool 94.19% <ø> (ø)
ceph 91.02% <ø> (ø)
cilium 85.84% <ø> (+1.88%) ⬆️
cisco_aci 95.88% <ø> (ø)
clickhouse 96.55% <ø> (ø)
cloud_foundry_api 95.98% <ø> (+0.12%) ⬆️
cockroachdb 97.18% <ø> (ø)
consul 93.04% <ø> (ø)
coredns 96.36% <ø> (ø)
couch 95.15% <ø> (+0.24%) ⬆️
couchbase 81.45% <ø> (ø)
crio 100.00% <ø> (ø)
datadog_checks_base 89.55% <90.44%> (+0.42%) ⬆️
datadog_checks_dev 80.49% <ø> (+0.10%) ⬆️
datadog_checks_downloader 80.40% <ø> (ø)
directory 94.70% <ø> (ø)
disk 91.00% <ø> (-0.51%) ⬇️
dns_check 94.00% <ø> (ø)
dotnetclr 100.00% <ø> (ø)
druid 97.70% <ø> (ø)
ecs_fargate 77.65% <ø> (ø)
eks_fargate 94.05% <ø> (ø)
elastic 88.54% <ø> (ø)
envoy 93.77% <ø> (+0.25%) ⬆️
etcd 93.09% <ø> (ø)
exchange_server 100.00% <ø> (ø)
external_dns 100.00% <ø> (ø)
fluentd 94.77% <ø> (ø)
gearmand 77.27% <ø> (+1.29%) ⬆️
gitlab 89.94% <ø> (ø)
gitlab_runner 90.32% <ø> (ø)
glusterfs 80.09% <ø> (+0.92%) ⬆️
go_expvar 91.95% <ø> (ø)
gunicorn 94.29% <ø> (+0.76%) ⬆️
haproxy 95.22% <ø> (+0.17%) ⬆️
harbor 91.58% <ø> (ø)
hazelcast 92.39% <ø> (ø)
hdfs_datanode 90.00% <ø> (ø)
hdfs_namenode 87.94% <ø> (ø)
http_check 89.96% <ø> (+1.82%) ⬆️
ibm_db2 93.87% <ø> (ø)
ibm_mq 90.19% <ø> (+0.18%) ⬆️
ibm_was 97.44% <ø> (ø)
iis 92.41% <ø> (ø)
istio 93.25% <ø> (+0.56%) ⬆️
kafka_consumer 81.15% <ø> (ø)
kong 92.21% <ø> (ø)
kube_apiserver_metrics 97.35% <ø> (ø)
kube_controller_manager 97.05% <ø> (ø)
kube_dns 98.85% <ø> (ø)
kube_metrics_server 100.00% <ø> (ø)
kube_proxy 100.00% <ø> (ø)
kube_scheduler 98.07% <ø> (ø)
kubelet 89.47% <ø> (ø)
kubernetes_state 89.69% <ø> (ø)
kyototycoon 85.96% <ø> (ø)
lighttpd 83.64% <ø> (ø)
linkerd 86.12% <ø> (+1.15%) ⬆️
linux_proc_extras 96.22% <ø> (ø)
mapr 84.18% <ø> (ø)
mapreduce 81.81% <ø> (-0.46%) ⬇️
marathon 83.12% <ø> (ø)
marklogic 95.32% <ø> (ø)
mcache 93.39% <ø> (ø)
mesos_master 89.68% <ø> (ø)
mesos_slave 93.63% <ø> (ø)
mongo 94.74% <ø> (+0.28%) ⬆️
mysql 84.87% <ø> (+0.15%) ⬆️
nagios 89.53% <ø> (ø)
network 77.76% <ø> (+1.00%) ⬆️
nfsstat 95.20% <ø> (ø)
nginx 95.11% <ø> (+0.93%) ⬆️
nginx_ingress_controller 98.30% <ø> (ø)
openldap 96.33% <ø> (ø)
openmetrics 97.14% <ø> (ø)
openstack 51.30% <ø> (ø)
openstack_controller 90.59% <ø> (ø)
oracle 92.99% <ø> (+0.63%) ⬆️
pdh_check 97.77% <ø> (ø)
pgbouncer 91.50% <ø> (ø)
php_fpm 89.95% <ø> (+0.43%) ⬆️
postfix 88.04% <ø> (ø)
postgres 92.27% <ø> (+0.08%) ⬆️
powerdns_recursor 95.93% <ø> (ø)
process 85.12% <ø> (+0.28%) ⬆️
prometheus 94.17% <ø> (ø)
proxysql 99.62% <ø> (ø)
rabbitmq 93.73% <ø> (ø)
redisdb 87.26% <ø> (ø)
rethinkdb 97.93% <ø> (ø)
riak 99.22% <ø> (ø)
riakcs 93.61% <ø> (ø)
sap_hana 93.04% <ø> (ø)
scylla 100.00% <ø> (ø)
snmp 91.57% <ø> (+0.04%) ⬆️
snowflake 94.18% <ø> (+0.58%) ⬆️
sonarqube 95.69% <ø> (ø)
spark 93.64% <ø> (ø)
sqlserver 81.74% <ø> (ø)
squid 100.00% <ø> (ø)
ssh_check 91.08% <ø> (ø)
statsd 87.36% <ø> (+1.05%) ⬆️
supervisord 92.30% <ø> (ø)
system_core 91.04% <ø> (ø)
system_swap 98.30% <ø> (ø)
tcp_check 87.95% <ø> (ø)
teamcity 80.00% <ø> (ø)
tls 97.04% <ø> (+0.87%) ⬆️
tokumx 58.40% <ø> (?)
twemproxy 78.33% <ø> (ø)
twistlock 80.74% <ø> (ø)
varnish 84.57% <ø> (+0.24%) ⬆️
vault 94.80% <ø> (+0.54%) ⬆️
vertica 92.30% <ø> (ø)
voltdb 96.81% <ø> (ø)
vsphere 89.77% <ø> (+0.08%) ⬆️
win32_event_log 86.03% <ø> (+0.28%) ⬆️
windows_service 95.83% <ø> (ø)
wmi_check 92.91% <ø> (ø)
yarn 90.30% <ø> (ø)
zk 85.17% <ø> (+0.75%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Adds a new `DBMAsyncJob` class which will be used by the `postgres` and `mysql` checks to schedule asynchronous tasks that need to run independently of the main check.

Follow-up PRs:
* postgres: #9657
* mysql: #9658
@djova djova marked this pull request as ready for review July 8, 2021 16:32
@djova djova requested review from a team as code owners July 8, 2021 16:32
@ofek ofek changed the title add db.utils.DBMAsyncJob Add db.utils.DBMAsyncJob Jul 8, 2021
@djova djova merged commit 0113b0a into master Jul 8, 2021
@djova djova deleted the djova/add-dbm-async-job branch July 8, 2021 18:37
djova added a commit that referenced this pull request Jul 8, 2021
* decouple the DBM metrics collection interval from the check run interval
* set default DBM metrics collection interval to 10s
* change `statement_samples.collections_per_second` to `statement_samples.collection_interval` so it matches the new `statement_metrics.collection_interval` key

Depends on #9656

Motivation: Being able to configure the DBM metrics collection interval separately from the check run interval enables us to use a 10 second interval (by default) for the query metrics. There are various difficulties when querying metrics that have a 15 second interval (i.e. ensuring a correct rollup window for varying time ranges) that don't exist with a 10 second interval.
djova added a commit that referenced this pull request Jul 8, 2021
* decouple the DBM metrics collection interval from the check run interval
* set default DBM metrics collection interval to 10s
* change `statement_samples.collections_per_second` to `statement_samples.collection_interval` so it matches the new `statement_metrics.collection_interval` key

Depends on #9656

Motivation: being able to configure the DBM metrics collection interval separately from the check run interval enables us to use a 10 second interval (by default) for the query metrics. There are various difficulties when querying metrics that have a 15 second interval (i.e. ensuring a correct rollup window for varying time ranges) that don't exist with a 10 second interval.
djova added a commit that referenced this pull request Jul 8, 2021
* decouple the DBM metrics collection interval from the check run interval
* set default DBM metrics collection interval to 10s
* change `statement_samples.collections_per_second` to `statement_samples.collection_interval` so it matches the new `statement_metrics.collection_interval` key

Depends on #9656

Motivation: being able to configure the DBM metrics collection interval separately from the check run interval enables us to use a 10 second interval (by default) for the query metrics. There are various difficulties when querying metrics that have a 15 second interval (i.e. ensuring a correct rollup window for varying time ranges) that don't exist with a 10 second interval.
djova added a commit that referenced this pull request Jul 8, 2021
* decouple the DBM metrics collection interval from the check run interval
* set default DBM metrics collection interval to 10s
* change `statement_samples.collections_per_second` to `statement_samples.collection_interval` so it matches the new `statement_metrics.collection_interval` key

Depends on #9656

Motivation: Being able to configure the DBM metrics collection interval separately from the check run interval enables us to use a 10 second interval (by default) for the query metrics. There are various difficulties when querying metrics that have a 15 second interval (i.e. ensuring a correct rollup window for varying time ranges) that don't exist with a 10 second interval.
djova added a commit that referenced this pull request Jul 9, 2021
* decouple DBM query metrics interval from check run interval

* decouple the DBM metrics collection interval from the check run interval
* set default DBM metrics collection interval to 10s
* change `statement_samples.collections_per_second` to `statement_samples.collection_interval` so it matches the new `statement_metrics.collection_interval` key

Depends on #9656

Motivation: being able to configure the DBM metrics collection interval separately from the check run interval enables us to use a 10 second interval (by default) for the query metrics. There are various difficulties when querying metrics that have a 15 second interval (i.e. ensuring a correct rollup window for varying time ranges) that don't exist with a 10 second interval.

* update config spec & model
djova added a commit that referenced this pull request Jul 9, 2021
* decouple DBM query metrics interval from check run interval

* decouple the DBM metrics collection interval from the check run interval
* set default DBM metrics collection interval to 10s
* change `statement_samples.collections_per_second` to `statement_samples.collection_interval` so it matches the new `statement_metrics.collection_interval` key

Depends on #9656

Motivation: Being able to configure the DBM metrics collection interval separately from the check run interval enables us to use a 10 second interval (by default) for the query metrics. There are various difficulties when querying metrics that have a 15 second interval (i.e. ensuring a correct rollup window for varying time ranges) that don't exist with a 10 second interval.

* update spec and model
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants